Skip to content

feat: weight edits read the same on every surface, and experiments record their checkpoint - #32

Merged
jackulau merged 21 commits into
mainfrom
feat/weights-all
Nov 12, 2026
Merged

jackulau merged 21 commits into
mainfrom
feat/weights-all

Conversation

@jackulau

Copy link
Copy Markdown
Owner

A weight edit now reads the same wherever it is recorded or shown, and an experiment says which checkpoint it ran.

What changes

  • Sessions, CLI, MCP, viewer. A session that recorded weight edits gets stats.weightEdits (edits, distinct tensors, weights changed) and the weights:edited tag, and is listed even when it holds nothing else. coding.sh show, MCP session_get and the viewer print one line per edit from one formatter (MCP used to drop these events). --where weights.edits and weights.changed filter sessions, in the SQLite index and in the Postgres and ClickHouse stores. SESSION_FOLD_VERSION is bumped, so stored sessions are folded again.
  • OpenTelemetry. A span or log record named weights.edit with coding_sh.weights.* attributes is read as the same event. The otlp sink sends one weights.edit span per edit, and [serve.forward] does for edits posted to /v1/ingest. Metadata mode keeps only the counts and norms.
  • TypeScript SDK. weightsEdit() and weightsDelta(), matching the Python SDK's weights_edit(). Both SDKs' tests check one shared fixture of the event.
  • Evals. exp run --weights <SHA256> stamps a run with its checkpoint. exp compare, exp show, exp list, the viewer and the MCP tools show the stamp, and a comparison warns when both sides ran the same weights or only one recorded them. exp metrics prints a small non-zero value with its digits instead of 0.000.
  • Python SDK fix. weights_edit() reads a 0-d tensor or array as the number it holds, so l2=delta.norm() records the edit.

Why one PR

This goes against "one change per PR" on purpose, as #24 did. The four branches (feat/weights-surfaces, feat/sdk-ts-weights, feat/exp-weights-stamp, fix/sdk-python-tensor-scalars) each touch CHANGELOG.md and docs/reference/api.md, and two limits only show once they are together (the MCP description byte budget and the changelog's 60-word rule). Each commit is still one change. Merge with a merge commit, not a squash.

Checks

On the merged branch, on macOS:

  • npm run check in packages/coding.sh: typecheck clean; the last root npm test had 6311 tests, 6299 pass, 1 fail, 11 skipped. The one failure is the timing test a 12-hour session in test/cli/longrun.test.ts; its file passes alone (11 of 11) and the run before the last small commit had 0 failures.
  • TS SDK 102 of 102. Python SDK 119 tests, 0 skipped with CODING_SH_SDK_REQUIRE_ALL=1, mypy clean.
  • npm run build, npm run lint: clean. The generated CLI reference, SDK reference and JSON Schema are not stale.
  • By hand, in a throwaway store: one edit sent by the Python SDK, the TS SDK and an OTLP span stores one identical payload and prints one identical line in show and in MCP session_get.

The local runs unset ANTHROPIC_BASE_URL, T3CODE_THREAD_ID, CLAUDE_CODE_ENTRYPOINT and CLAUDE_CODE_SESSION_ID. With them set, test/capture/hook.test.ts and test/cli/rebuild-redactions.test.ts fail, on main too: the tests pass the caller's environment to hook.sh. A follow-up makes them hermetic.

Not checked locally

  • The new store filter keys against a live Postgres or ClickHouse (CI runs the ClickHouse job).
  • The Python fix against real torch; it was checked with numpy scalars, 0-d arrays and a stand-in class.

Notes for review

  • The MCP tool-surface byte budget in test/mcp/toolevals.test.ts goes from 43,900 to 44,100 for the weights output fields. Descriptions stay under their own 11,850 limit.
  • Follow-ups in progress: a link from an experiment's checkpoint to the session whose edits made it, and forwarding edits that arrive as OTLP logs.

…ine in the CLI, MCP and viewer

A weights.edit notification was only stored: show printed a blank
notification row, MCP's session_get left it out of the timeline, and no
stat or tag said a session had edited weights.

Sessions now carry stats.weightEdits (events, tensors, weightsChanged)
and the weights:edited tag, with --where keys weights.edits and
weights.changed. show, MCP (sessions_list, session_get, the session
resource) and the viewer write one line per edit from one formatter. A
session that holds only a weight edit is no longer hidden as empty.
SESSION_FOLD_VERSION 20 refolds stored sessions.
Transport.enqueue(), the client's enqueue() and emit() return true when
the item was queued and false when it was dropped or the SDK is off or
shut down. Nothing reads the result yet; weightsEdit() will.
…ord the same weight edit as Python

weightsEdit() sends one notification event with payload.kind
"weights.edit", with the same keys, values, validation and session
shape as the Python SDK's weights_edit(). It never throws and returns
true only when the event was queued. weightsDelta() computes shape,
changed, l2 and maxAbs for nested arrays and typed arrays.

packages/sdk/test/fixtures/weights-edit.json holds the events an edit
must send, recorded from the Python SDK. Both test suites read it.
… or leaves through the otlp sink

A span or a log record named weights.edit with coding_sh.weights.*
attributes is now read as the weights.edit notification the SDK sends,
so the session gets the same stat, tag and line. Before, a span became
a generic span and a root span lost its attributes.

The otlp sink and serve's forward send one weights.edit span per edit
with the same attributes, and the invoke_agent span carries
coding_sh.stats.weight_edits and coding_sh.stats.weights_changed.
Metadata mode keeps the counts and the norms: the forward allowlist
drops the tensor name, shape, hashes and note.
…nd an impossible shape or count is not printed as a total

MCP's session_get dropped an edit whose payload had no well-formed
field. The line no longer says "3 of 0 weights" when the count does not
fit in the shape, reads no shape of more than 16 dimensions (the SDK's
limit), and never rounds a norm to Infinity. The docs now say which
edits serve's forward sends.
…g the tool surface

The words added for weight edits are paid for by a shorter opening and
a shorter note on the turn argument, so the description budget still
holds when other tools add words.
…nd exp compare names both

exp run --weights <SHA256> stores metadata.weights = { sha256 } (64 hex, lowercased; coding.sh
hashes nothing). exp compare shows each side's stamp in the normal and the --given header, warns
when both sides ran the same weights and neither regrades the other, and says so when only one
side is stamped. exp show, the exp run summary, the viewer and the MCP tools experiments_list and
experiment_compare carry the same facts.

--arm takes one stamp per arm (--weights base=<SHA> --weights edited=<SHA>). --watch and matrix
specs refuse the flag. --resume keeps the stored stamp and refuses a different one; the stamp is
part of the job config. A regrade inherits the stamp.
…, not 0.000

A count or score below 0.0005 (a gain per edited weight of 0.0001) printed as 0.000 and read as
zero. It now prints two significant digits; values that three decimals already show are unchanged.
The Metric A/B guide names three keys for an arm that reports a model edit (edited_weights,
gain_pp, gain_per_weight), says the two gain keys need --higher, and shows the table. A CLI test
runs a stub arm that prints them and checks the three rows and their direction.
…ts and weights.changed

The two --where keys had SQL only in the SQLite store, so a query that
used them was refused on the other two.
l2=delta.norm() in torch is a 0-d tensor, not a Python number, so the
edit was refused and dropped with a warning while the docs said torch
scalars count. A value with ndim 0 and item() is now read through
item(); a tensor of more than one value is still refused.
# Conflicts:
#	CHANGELOG.md
#	docs/reference/api.md
@jackulau
jackulau merged commit 5ceb013 into main Nov 12, 2026
28 checks passed
@jackulau
jackulau deleted the feat/weights-all branch November 12, 2026 22:22
jackulau added a commit that referenced this pull request Nov 12, 2026
…rning is once per process

Four follow-ups to #32, one per commit.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant